[Python-Dev] How to stay almost backwards compatible with all these new cool features
Here's some of my ideas about subject. Maybe some of them are rather
foolish, others -- rather simple and common... I just want to add my 2
cents to Python development.
1) What is the reason for making Python backwards incompatible (let it
be just 'BIC', and let 'BC' stands for 'backwards compatible')? The
reason is revolution. But how much can we get just from intensive
evolution?
2) Is there any way both for staying (almost) BC and intense evolving?
Yes. General rule is rather simple: make old way of doing something
*deprecated* (but *not* remove it entirely) and *add* the new way of
doing this. But how to _force_ users to use new way instead of old? My
proposal: Python should raise DeprecationError after old way is used:
old_way()
should be equivalent to
old_way()
raise DeprecationError('description')
So people who want to use old way should write something like
try:
old_way()
except DeprecationError:
pass
I think they soon will migrate to new style :-)
[Manual/semi-automatic migration]
Another benefit is that apps that were written for old-way Python
version could be run under new-way Python after this simple
modification (there might be standard script for making old apps
compatible with new versions of Python!). [Automatic migration]
3) Staying BC means no revolutions in syntax. But there are problems
with some of new-style features:
a) 'raise' statement.
I dislike idea about inheriting all exceptions from the base class and
about removing 'raise' in favor of '.raise()'. Reasons: we can think
about 'raise' as about more powerful variant of 'return'. Classic
example is recursive search in binary tree: raising the result there
seems to be very elegant and agile. Exception != Error is true IMHO.
b) Interfaces.
I like them. But isn't it ugly to write:
interface SuperInterface: ...
Note: 'interface' is repeated there two times! And this is *not* BC
solution at all. Remember exception classes:
class MyCommonError(Exception): ...
but not
exception MyCommonError(...): ...
and it seems to be OK! And I have great agility with it: as mentioned,
I can raise just 'some_object', but not only 'exception'.
So, my proposal is syntax
class SuperInterface(Interface): ...
or maybe
class SuperInterface: ...
but not
interface SuperInterface: ...
Note: first two variants are BC solutions!
And *yes*, you should be able to implement *any* class. Example:
class Fish(object):
def swim():
do_swim()
# else ...
class Dog(object):
def bark():
do_bark()
# else ...
class SharkLikeDog(Fish) implements Dog
Isn't it very good-looking?
Note: IMHO Type == Implemented interface. So that's why every
type/class can be used as interface. (Sorry for type/class mess).
Could we find some benefits of it?
c) I like Optional TypeChecking. But I think it could be improved with
implementing some sort of Optional InterfaceChecking. Maybe like this:
def f(a implements B, c: D = 'HELLO') implements E:
# ?function implements interface? well, maybe it can be some type
check interface?
# some code here
or
def f(a implements B, c: D = 'HELLO') -> implements E:
# some code here
Summary
--
Well, I think the main idea is (2):
- Don't remove; make it strongly deprecated
Then:
- Some changes to interfaces implementation
- etc ('raise' statement, InterfaceCheck -- see (3) )
Sorry for my English and for mess.
-- Regards, Gregory.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] s/hotshot/lsprof
Hi! The current Python profilers situation is a mess. 'profile.Profile' is the ages-old pure Python profiler. At the end of a run, it builds a dict that is inspected by 'pstats.Stats'. It has some recent support for profiling C calls, which however make it crash in some cases [1]. And of course it's slow (makes a run take about 10x longer). 'hotshot', new from 2.2, is quite faster (reportedly, only 30% added overhead). The log file is then loaded and turned into an instance of the same 'pstats.Stats'. This loading takes ages. The reason is that the log file only records events, and loading is done by instantiating a 'profile.Profile' and sending it all the events. In other words, it takes exactly as long as the time it spared in the first place! Moreover, for some reasons, the results given by hotshot seem sometimes quite wrong. (I don't understand why, but I've seen it myself, and it's been reported by various people, e.g. [2].) 'hotshot' doesn't know about C calls, but it can log line events, although this information is lost(!) in the final conversion to a 'pstats.Stats'. 'lsprof' is a third profiler by Brett Rosen and Ted Czotter, posted on SF in June [2]. Michael Hudson and me did some minor clean-ups and improvements on it, and it seems to be quite useful. It is, for example, the only of the three profilers that managed to give sensible information about the PyPy translation process without crashing, allowing us to accelerate it from over 30 to under 20 minutes. The SF patch contains a more detailed account on the reasons for writing 'lsprof'. The current version [3] does not support C calls nor line events. It has its own simple interface, which is not compatible with any of the other two profilers. However, unlike the other two profilers, it can record detailed stats about children, which I found quite useful (e.g. how much take is spent in a function when it is called by another specific function). Therefore, I think it would be a great idea to add 'lsprof' to the standard library. Unless there are objections, it seems that the best plan is to keep 'profile.py' as a pure Python implementation and replace 'hotshot' with 'lsprof'. Indeed, I don't see any obvious advantage that 'hotshot' has over 'lsprof', and I certainly see more than one downside. Maybe someone has a use for (and undocumented ways to fish for) line events generated by hotshot. Well, there is a script [4] to convert hotshot log files to some format that a KDE tool [5] can display. (It even looks like hotshot files were designed with this in mind.) Given that the people doing that can still compile 'hotshot' as a separate extension module, it doesn't strike me as a particularly good reason to keep Yet Another Profiler in the standard library. So here is my plan: Unify a bit more the interfaces of the pure Python and the C profilers. This also means that 'lsprof' should be made to use a pstats-compatible log format. The 'pstats' documentation specifically says that the file format can change: that would give 'lsprof' a place to store its detailed children stats. Then we can provide a dummy 'hotshot.py' for compatibility, remove its documentation, and provide documentation for 'lsprof'. If anyone feels like this is a bad idea, please speak up. A bientot, Armin [1] https://sourceforge.net/tracker/?group_id=5470&atid=105470&func=detail&aid=1117670 [2] http://sourceforge.net/tracker/?group_id=5470&atid=305470&func=detail&aid=1212837 [3] http://codespeak.net/svn/user/arigo/hack/misc/lsprof (Subversion) [4] http://mail.python.org/pipermail/python-list/2003-September/183887.html [5] http://kcachegrind.sourceforge.net/cgi-bin/show.cgi ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] str.dedent
On 11/18/05, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> Noam Raphael wrote:
> > I just wanted to add another use case: long messages. Consider those
> > lines from idlelib/run.py:133
> >
> > msg = "IDLE's subprocess can't connect to %s:%d. This may be due "\
> > "to your personal firewall configuration. It is safe to "\
> > "allow this internal connection because no data is visible on
> > "\
> > "external ports." % address
> > tkMessageBox.showerror("IDLE Subprocess Error", msg, parent=root)
>
> You are missing an important point here: There are intentionally no line
> breaks in this string; it must be a single line, or else showerror will
> break it in funny ways. So converting it to a multi-line string would
> break it, dedent or not.
Only if you didn't include newline escapes, e.g.::
msg = textwrap.dedent('''\
IDLE's subprocess can't connect to %s:%d. This may be due \
to your personal firewall configuration. It is safe to \
allow this internal connection because no data is visible on \
external ports.''' % address)
STeVe
--
You can wordify anything if you just verb it.
--- Bucky Katt, Get Fuzzy
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] str.dedent
On 11/19/05, Steven Bethard <[EMAIL PROTECTED]> wrote:
> > You are missing an important point here: There are intentionally no line
> > breaks in this string; it must be a single line, or else showerror will
> > break it in funny ways. So converting it to a multi-line string would
> > break it, dedent or not.
>
> Only if you didn't include newline escapes, e.g.::
>
> msg = textwrap.dedent('''\
> IDLE's subprocess can't connect to %s:%d. This may be due \
> to your personal firewall configuration. It is safe to \
> allow this internal connection because no data is visible on \
> external ports.''' % address)
>
Unfortunately, it won't help, since the 'dedent' method won't treat
those spaces as indentation.
But if those messages were printed to the standard error, the line
breaks would be ok, and the use case valid.
Noam
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Patch Req. # 1351020 & 1351036: PythonD modifications
[EMAIL PROTECTED] wrote: > I would appreciate feedback concerning these patches before the next > "PythonD" (for DOS/DJGPP) is released. PEP 11 says that DOS is not supported anymore since Python 2.0. So I am -1 on reintroducing support for it. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How to stay almost backwards compatible with all these new cool features
On Sat, Nov 19, 2005, Gregory Petrosyan wrote: > > Here's some of my ideas about subject. Maybe some of them are rather > foolish, others -- rather simple and common... I just want to add my 2 > cents to Python development. This message was more appropriate for comp.lang.python; most of what you talk about has already been discussed before, and the rest has to do with user-level changes. Please continue this discussion there if you're interested in the subject. Thank you. -- Aahz ([EMAIL PROTECTED]) <*> http://www.pythoncraft.com/ "If you think it's expensive to hire a professional to do the job, wait until you hire an amateur." --Red Adair ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] s/hotshot/lsprof
On Sat, Nov 19, 2005, Armin Rigo wrote: > > If anyone feels like this is a bad idea, please speak up. This sounds like a good idea, and your presentation already looks almost like a PEP. How about going ahead and making it a formal PEP, which will make it easier to push through the dev process? -- Aahz ([EMAIL PROTECTED]) <*> http://www.pythoncraft.com/ "If you think it's expensive to hire a professional to do the job, wait until you hire an amateur." --Red Adair ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] s/hotshot/lsprof
Armin Rigo wrote: > If anyone feels like this is a bad idea, please speak up. As stated, it certainly is a bad idea. To make it a good idea, there should also be some commitment to maintain this library for a number of years. So who would be maintaining it, and what are their plans for doing so? Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] s/hotshot/lsprof
Just for everyone's FYI while we are talking about profilers, Floris Bruynooghe (who I am cc'ing on this so he can contribute to the conversation), for Google's Summer of Code, wrote a replacement for 'profile' that uses Hotshot directly. Thanks to his direct use of Hotshot and rewrite of pstats it loads Hotshot data 30% faster and also alleviates keeping 'profile' around and its slightly questionable license. You can find his project at http://savannah.nongnu.org/projects/pyprof/ . I believe he also tweaked Hotshot to accept custom timing functions. I have not had a chance to go over his code to clean it up for putting it up on SF, but I thought people should be aware of it. -Brett On 11/19/05, Armin Rigo <[EMAIL PROTECTED]> wrote: > Hi! > > The current Python profilers situation is a mess. > > 'profile.Profile' is the ages-old pure Python profiler. At the end of a > run, it builds a dict that is inspected by 'pstats.Stats'. It has some > recent support for profiling C calls, which however make it crash in > some cases [1]. And of course it's slow (makes a run take about 10x > longer). > > 'hotshot', new from 2.2, is quite faster (reportedly, only 30% added > overhead). The log file is then loaded and turned into an instance of > the same 'pstats.Stats'. This loading takes ages. The reason is that > the log file only records events, and loading is done by instantiating a > 'profile.Profile' and sending it all the events. In other words, it > takes exactly as long as the time it spared in the first place! > Moreover, for some reasons, the results given by hotshot seem sometimes > quite wrong. (I don't understand why, but I've seen it myself, and it's > been reported by various people, e.g. [2].) 'hotshot' doesn't know > about C calls, but it can log line events, although this information is > lost(!) in the final conversion to a 'pstats.Stats'. > > 'lsprof' is a third profiler by Brett Rosen and Ted Czotter, posted on > SF in June [2]. Michael Hudson and me did some minor clean-ups and > improvements on it, and it seems to be quite useful. It is, for > example, the only of the three profilers that managed to give sensible > information about the PyPy translation process without crashing, > allowing us to accelerate it from over 30 to under 20 minutes. The SF > patch contains a more detailed account on the reasons for writing > 'lsprof'. The current version [3] does not support C calls nor line > events. It has its own simple interface, which is not compatible with > any of the other two profilers. However, unlike the other two > profilers, it can record detailed stats about children, which I found > quite useful (e.g. how much take is spent in a function when it is > called by another specific function). > > Therefore, I think it would be a great idea to add 'lsprof' to the > standard library. Unless there are objections, it seems that the best > plan is to keep 'profile.py' as a pure Python implementation and replace > 'hotshot' with 'lsprof'. Indeed, I don't see any obvious advantage that > 'hotshot' has over 'lsprof', and I certainly see more than one downside. > Maybe someone has a use for (and undocumented ways to fish for) line > events generated by hotshot. Well, there is a script [4] to convert > hotshot log files to some format that a KDE tool [5] can display. (It > even looks like hotshot files were designed with this in mind.) Given > that the people doing that can still compile 'hotshot' as a separate > extension module, it doesn't strike me as a particularly good reason to > keep Yet Another Profiler in the standard library. > > So here is my plan: > > Unify a bit more the interfaces of the pure Python and the C profilers. > This also means that 'lsprof' should be made to use a pstats-compatible > log format. The 'pstats' documentation specifically says that the file > format can change: that would give 'lsprof' a place to store its > detailed children stats. > > Then we can provide a dummy 'hotshot.py' for compatibility, remove its > documentation, and provide documentation for 'lsprof'. > > If anyone feels like this is a bad idea, please speak up. > > > A bientot, > > Armin > > > [1] > https://sourceforge.net/tracker/?group_id=5470&atid=105470&func=detail&aid=1117670 > > [2] > http://sourceforge.net/tracker/?group_id=5470&atid=305470&func=detail&aid=1212837 > > [3] http://codespeak.net/svn/user/arigo/hack/misc/lsprof (Subversion) > > [4] http://mail.python.org/pipermail/python-list/2003-September/183887.html > > [5] http://kcachegrind.sourceforge.net/cgi-bin/show.cgi > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: h
Re: [Python-Dev] ast status, memory leaks, etc
I lied a bit in my previous status. I said that the refs used at the end of a regression test run from a clean state (*) were down to 380k. Well if I had remembered to remove all the .pyc's this would have been true. Here's the numbers now: Before AST: r39757 [362766 refs] Before AST: svn up [356255 refs] 266 OK 31 skipped clean: [342367 refs] 267 OK 31 skipped (*) Before each run I did: find . -name '*.pyc' | xargs rm Unless I screwed up again, the first line is from clean at revision 39757 which was just before the AST merge. The second line was a selective update of other files that didn't have any relationship to AST (primarily compile.c and symtable.c). The last run is after my recent checkin. So even with an additional test, we are finishing a regrtest.py run with less references. I don't know of any constructs which leak references. A patch was posted for the free memory read I reported earlier (not related to AST branch). It's on SF, I don't know the #. There are many potential memory leaks in the AST code in error conditions (hopefully these are only possible when running out of memory). It really needs the arena implementation to fix them and get it right. There are also still a few printfs in the AST code which should be changed to SystemErrors. There are still 2 memory leaks while running the regression tests. They show up when running test_fork1 and test_pty. There may be more, valgrind crashed on me the last run which was also before I fixed some of the reference leaks. It would be great if people could localize the leaks. 512 bytes in 1 blocks are definitely lost in loss record 319 of 548 at 0x11B1AF13: malloc (vg_replace_malloc.c:149) by 0x433CC4: new_arena (obmalloc.c:500) by 0x433EA8: PyObject_Malloc (obmalloc.c:706) by 0x43734B: PyString_FromStringAndSize (stringobject.c:74) by 0x4655B5: optimize_code (compile.c:957) by 0x467B86: makecode (compile.c:4092) by 0x467F00: assemble (compile.c:4166) by 0x46AA94: compiler_mod (compile.c:1755) by 0x46AC8B: PyAST_Compile (compile.c:285) by 0x47A870: run_mod (pythonrun.c:1195) by 0x47B0E8: PyRun_StringFlags (pythonrun.c:1159) by 0x45767A: builtin_eval (bltinmodule.c:589) by 0x41684F: PyObject_Call (abstract.c:1777) by 0x45EB4B: PyEval_CallObjectWithKeywords (ceval.c:3432) by 0x457E4E: builtin_map (bltinmodule.c:938) 1280 bytes in 2 blocks are definitely lost in loss record 383 of 548 at 0x11B1AF13: malloc (vg_replace_malloc.c:149) by 0x433CC4: new_arena (obmalloc.c:500) by 0x433EA8: PyObject_Malloc (obmalloc.c:706) by 0x4953F3: PyNode_AddChild (node.c:95) by 0x495611: shift (parser.c:112) by 0x4958F0: PyParser_AddToken (parser.c:244) by 0x411704: parsetok (parsetok.c:166) by 0x47AE4F: PyParser_ASTFromFile (pythonrun.c:1292) by 0x472338: parse_source_module (import.c:777) by 0x47262B: load_source_module (import.c:905) by 0x4735B3: load_module (import.c:1665) by 0x473C4B: import_submodule (import.c:2259) by 0x473DE0: load_next (import.c:2079) by 0x4741D5: import_module_ex (import.c:1914) by 0x474389: PyImport_ImportModuleEx (import.c:1955) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
