Re: [Python-Dev] ctypes and win64

2006-10-20 Thread Thomas Heller
Thomas Heller schrieb (this was before Python 2.5 had been released):
> The _ctypes extension module does currently not even build on Win64.
> 
> I'm (slowly) working on this (for AMD64, not for itanium), but it may
> take a good while before it is stable - It is not even fully implemented
> currently.
> 
> The win64 msi installer installs the ctypes package anyway, but it cannot be
> imported.
> 
> I suggest that it should be removed from the 2.5 win64 msi installers, so that
> at least, when it is ready, can be installed as separate package.


Then, Martin changed the win64 msi installer to exclude the ctypes package
when the _ctypes.pyd extension does not exist because it was not built.

In the meantime I have integrated patches (in the trunk) so that _ctypes
can be built for win64/AMD64, and does even work.

Can these changes be merged into release25-maint?  IMO this is low-risk
because they contain only small changes to the files in 
Modules/_ctypes/libffi_msvc,
plus *some* changes to support the Windows LP64 model.

I would prefer to merge these changes into release25-maint, because I want to
also release the standalone ctypes packages from this branch (using it with
svn:externals from somewhere else).

The official Python 2.5.x win64/AMD64 windows installers should still *not*
contain the ctypes package, but they could install it separately.

Thanks,
Thomas

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] ctypes and win64

2006-10-20 Thread Thomas Heller
[Resent after subscribing to python-dev with this new email address,
sorry if it appears twice]

Thomas Heller schrieb (this was before Python 2.5 had been released):
> > The _ctypes extension module does currently not even build on Win64.
> > 
> > I'm (slowly) working on this (for AMD64, not for itanium), but it may
> > take a good while before it is stable - It is not even fully implemented
> > currently.
> > 
> > The win64 msi installer installs the ctypes package anyway, but it cannot be
> > imported.
> > 
> > I suggest that it should be removed from the 2.5 win64 msi installers, so 
> > that
> > at least, when it is ready, can be installed as separate package.


Then, Martin changed the win64 msi installer to exclude the ctypes package
when the _ctypes.pyd extension does not exist because it was not built.

In the meantime I have integrated patches (in the trunk) so that _ctypes
can be built for win64/AMD64, and does even work.

Can these changes be merged into release25-maint?  IMO this is low-risk
because they contain only small changes to the files in 
Modules/_ctypes/libffi_msvc,
plus *some* changes to support the Windows LP64 model.

I would prefer to merge these changes into release25-maint, because I want to
also release the standalone ctypes packages from this branch (using it with
svn:externals from somewhere else).

The official Python 2.5.x win64/AMD64 windows installers should still *not*
contain the ctypes package, but they could install it separately.

Thanks,
Thomas

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Python unit tests failing on Pybots farm

2006-10-20 Thread Grig Gheorghiu
On 10/19/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> Grig Gheorghiu schrieb:
> > OK, I deleted the checkout directory on one of my buidslaves and
> > re-ran the build steps. The tests passed. So my conclusion is that a
> > full rebuild is needed for the tests to pass after the last checkins
> > (which included files such as configure and configure.in).
>
> Indeed, you had to re-run configure. There was a bug where -Werror was
> added to the build flags, causing several configure tests to fail
> (most notably, it would determine that there's no memmove on Linux).
>
> > Maybe the makefiles should be modified so that a full rebuild is
> > triggered when the configure and configure.in files are changed?
>
> The makefiles already do that: if configure changes, a plain
> "make" will first re-run configure.

Well, that didn't trigger a full rebuild on the Pybots buildslaves though.

>
> > At this point, I'll have to tell all the Pybots owners to delete their
> > checkout directories and start a new build.
>
> Not necessarily. You can also ask, at the buildbot GUI, that a
> non-existing branch is build. This should cause the checkouts
> to be deleted (and then the build to fail); the next regular
> build will check out from scratch.
>

OK, I'll try that next time. Or I can add an extra 'clean checkout
dir' step to the buildmaster -- but that would trigger a full rebuild
every time, which is not what I want, since some of the buildslaves
take a long time to do that.

Grig
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Python unit tests failing on Pybots farm

2006-10-20 Thread Martin v. Löwis
Grig Gheorghiu schrieb:
>> > Maybe the makefiles should be modified so that a full rebuild is
>> > triggered when the configure and configure.in files are changed?
>>
>> The makefiles already do that: if configure changes, a plain
>> "make" will first re-run configure.
> 
> Well, that didn't trigger a full rebuild on the Pybots buildslaves though.

Can you provide more details? Did it not run configure again, or
did that not cause a rebuild?

There is an issue with setup.py/distutils not doing the rebuilding
properly if header files change; contributions to fix this are welcome
(quick-hacked work-arounds are not).

Regards,
Martin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] ctypes and win64

2006-10-20 Thread Martin v. Löwis
Thomas Heller schrieb:
> I would prefer to merge these changes into release25-maint, because I want to
> also release the standalone ctypes packages from this branch (using it with
> svn:externals from somewhere else).

That's not a good reason for back-porting. If you want a "maintenance"
branch for ctypes, feel free to create one in the subversion, likewise
for tags.

OTOH, I can't comment on whether those changes would be acceptable for
a backport to the 2.5 maintenance branch - if they don't introduce
actual new features, it might be ok.

> The official Python 2.5.x win64/AMD64 windows installers should still *not*
> contain the ctypes package, but they could install it separately.

I don't really understand. Are you planning to back-port PCbuild changes
also? If so, how should including those extensions be suppressed?

Regards,
Martin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Python unit tests failing on Pybots farm

2006-10-20 Thread Grig Gheorghiu
On 10/20/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> Grig Gheorghiu schrieb:
> >> > Maybe the makefiles should be modified so that a full rebuild is
> >> > triggered when the configure and configure.in files are changed?
> >>
> >> The makefiles already do that: if configure changes, a plain
> >> "make" will first re-run configure.
> >
> > Well, that didn't trigger a full rebuild on the Pybots buildslaves though.
>
> Can you provide more details? Did it not run configure again, or
> did that not cause a rebuild?
>
> There is an issue with setup.py/distutils not doing the rebuilding
> properly if header files change; contributions to fix this are welcome
> (quick-hacked work-arounds are not).
>


Here are the steps that led to the unit test failures, after your
checkin of configure and configure.in.

svn update: 
http://www.python.org/dev/buildbot/community/all/x86%20Ubuntu%20Breezy%20trunk/builds/55/step-svn/0

configure: 
http://www.python.org/dev/buildbot/community/all/x86%20Ubuntu%20Breezy%20trunk/builds/55/step-configure/0

compile: 
http://www.python.org/dev/buildbot/community/all/x86%20Ubuntu%20Breezy%20trunk/builds/55/step-compile/0

test: 
http://www.python.org/dev/buildbot/community/all/x86%20Ubuntu%20Breezy%20trunk/builds/55/step-test/0


HTH,

Grig
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Python unit tests failing on Pybots farm

2006-10-20 Thread Martin v. Löwis
Grig Gheorghiu schrieb:
> Here are the steps that led to the unit test failures, after your
> checkin of configure and configure.in.
> 
> svn update: 
> http://www.python.org/dev/buildbot/community/all/x86%20Ubuntu%20Breezy%20trunk/builds/55/step-svn/0
> 
> configure: 
> http://www.python.org/dev/buildbot/community/all/x86%20Ubuntu%20Breezy%20trunk/builds/55/step-configure/0
> 
> compile: 
> http://www.python.org/dev/buildbot/community/all/x86%20Ubuntu%20Breezy%20trunk/builds/55/step-compile/0
> 
> test: 
> http://www.python.org/dev/buildbot/community/all/x86%20Ubuntu%20Breezy%20trunk/builds/55/step-test/0

As you can see, it indeed re-ran configure, and it also rebuilt the
interpreter. It then did not rebuild any of the extensions except for
pyexpat and elementtree.

As I said, contributions to fix that are welcome.

Regards,
Martin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] ctypes and win64

2006-10-20 Thread Thomas Heller
Martin v. Löwis schrieb:
> Thomas Heller schrieb:

[I was talking about patches to make ctypes work on 64-bit windows]

>> I would prefer to merge these changes into release25-maint, because I want to
>> also release the standalone ctypes packages from this branch (using it with
>> svn:externals from somewhere else).
> 
> That's not a good reason for back-porting. If you want a "maintenance"
> branch for ctypes, feel free to create one in the subversion, likewise
> for tags.
> 
> OTOH, I can't comment on whether those changes would be acceptable for
> a backport to the 2.5 maintenance branch - if they don't introduce
> actual new features, it might be ok.
> 
>> The official Python 2.5.x win64/AMD64 windows installers should still *not*
>> contain the ctypes package, but they could install it separately.
> 
> I don't really understand. Are you planning to back-port PCbuild changes
> also? If so, how should including those extensions be suppressed?

Let me try to put it in different words.

The official Python-2.5.amd64.msi does *not* contain ctypes, so
the official Python-2.5.x.amd64.msi should also not contain ctypes (I assume).

Not many people (I assume again) are running 64-bit windows, and use the 64-bit 
Python
version - but that will probably change soon.

I would like to merge the 64-bit windows related ctypes changes in trunk, as 
soon as
I'm sure that they work, back into the release25-maint branch.  And also make 
separate
ctypes releases from the release25-maint source code.  I will only backport 
these changes
if I'm convinced that they do not change the functionality of tehe current code.

This way win64 Python users could install ctypes from the separate release.
Also this way the source code for ctypes in the separate and the Python bundled
releases are exactly the same, without creating too much work because of the
different repositories.

Hope that makes the plan clear,
Thomas

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] OT: fdopen on Windows question

2006-10-20 Thread skip

Sorry for the off-topic post.  I figured someone here would know the answer
and I don't have access to Windows to check experimentally.

The ocrad program opens its input like so:

if ( std::strcmp( infile_name, "-" ) == 0 )
infile = stdin;
else
infile = std::fopen( infile_name, "r" );

(SpamBayes is starting to use ocrad and PIL to extract text from image
spam).  Ocrad fails on Windows because the input file is opened in text
mode.  That "r" should be "rb".  What's not clear to me is whether we can do
anything about stdin.  Will this work:

if ( std::strcmp( infile_name, "-" ) == 0 )
infile = std::fdopen( std::fileno(stdin), "rb" );
else
infile = std::fopen( infile_name, "rb" );

That is, can I change stdin from text to binary this way or is it destined
to always be in text mode?

Thx,

Skip
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] Massive test_sqlite failure on Mac OSX ... sometimes

2006-10-20 Thread skip
I'm setting up a buildbot slave for sqlalchemy on one of my Macs at home.
When it builds and tests Python's test suite the sqlite test fails.  When I
ran it alone like this:

./python.exe Lib/test/test_sqlite.py

and

./python.exe Lib/test/regrtest.py test_sqlite

it succeeded.  When I ran the full test suite it failed.  I then tried
adding -v as the error message suggested.  It hung in test_pty waiting for a
child process to complete.  (Is this a known problem?)  I finally redirected
stdout and stderr like so:

./python.exe Lib/test/regrtest.py -l -v > test.out 2>&1

and it completed.  It failed 146 out of 167 tests.  Here is a sample of the
failure messages:

...
CheckClose (sqlite3.test.dbapi.ConnectionTests) ... ERROR
CheckCommit (sqlite3.test.dbapi.ConnectionTests) ... ERROR
CheckCommitAfterNoChanges (sqlite3.test.dbapi.ConnectionTests) ... ERROR
CheckCursor (sqlite3.test.dbapi.ConnectionTests) ... ERROR
CheckExceptions (sqlite3.test.dbapi.ConnectionTests) ... ERROR
CheckFailedOpen (sqlite3.test.dbapi.ConnectionTests) ... ERROR
CheckRollback (sqlite3.test.dbapi.ConnectionTests) ... ERROR
CheckRollbackAfterNoChanges (sqlite3.test.dbapi.ConnectionTests) ... ERROR
CheckArraySize (sqlite3.test.dbapi.CursorTests) ... ERROR
CheckClose (sqlite3.test.dbapi.CursorTests) ... ERROR
CheckCursorConnection (sqlite3.test.dbapi.CursorTests) ... ERROR
CheckCursorWrongClass (sqlite3.test.dbapi.CursorTests) ... ERROR
CheckExecuteArgFloat (sqlite3.test.dbapi.CursorTests) ... ERROR
CheckExecuteArgInt (sqlite3.test.dbapi.CursorTests) ... ERROR
CheckExecuteArgString (sqlite3.test.dbapi.CursorTests) ... ERROR
CheckExecuteDictMapping (sqlite3.test.dbapi.CursorTests) ... ERROR
CheckExecuteDictMappingNoArgs (sqlite3.test.dbapi.CursorTests) ... ERROR
CheckExecuteDictMappingTooLittleArgs (sqlite3.test.dbapi.CursorTests) ... 
ERROR
CheckExecuteDictMappingUnnamed (sqlite3.test.dbapi.CursorTests) ... ERROR
CheckExecuteIllegalSql (sqlite3.test.dbapi.CursorTests) ... ERROR
CheckExecuteManyGenerator (sqlite3.test.dbapi.CursorTests) ... ERROR
CheckExecuteManyIterator (sqlite3.test.dbapi.CursorTests) ... ERROR
CheckExecuteManyNotIterable (sqlite3.test.dbapi.CursorTests) ... ERROR
...

A quick check of the tracebacks shows all the errors are of this form
(CheckClose is the first failure):

==
ERROR: CheckClose (sqlite3.test.dbapi.ConnectionTests)
--
Traceback (most recent call last):
  File 
"/Library/Buildbot/pybot/trunk.montanaro-g5/build/Lib/sqlite3/test/dbapi.py", 
line 85, in setUp
self.cx = sqlite.connect(":memory:")
ProgrammingError: library routine called out of sequence

That is, they all raise the same exception and all exceptions are raised on
sqlite.connect(":memory:") calls.  Sometimes there is a second parameter to
the call.

Anybody seen this before?

Skip
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] svn.python.org down

2006-10-20 Thread Martin v. Löwis
Grig Gheorghiu schrieb:
> FYI -- can't do svn checkouts/updates from the trunk at this point.
> 
> starting svn operation
> svn update --revision HEAD
>  in dir /home/twistbot/pybot/trunk.gheorghiu-x86/build (timeout 1200 secs)
> svn: PROPFIND request failed on '/projects/python/trunk'
> svn: PROPFIND of '/projects/python/trunk': could not connect to server
> (http://svn.python.org)

It turns out that there was a power surge at the colocation site where
the machines are, and, due to an unfortunate series of wreck, power went
out for about one second. When power came back, the machine rebooted,
but, for some reason, the svn apache server did not.

Regards,
Martin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] The "lazy strings" patch [was: PATCH submitted: Speed up + for string concatenation, now as fast as "".join(x) idiom]

2006-10-20 Thread Larry Hastings





I've significantly enhanced my string-concatenation patch, to the point
where that name is no longer accurate.  So I've redubbed it the "lazy
strings" patch.

The major new feature is that string *slices* are also represented with
a lazy-evaluation placeholder for the actual string, just as
concatenated strings were in my original patch.  The lazy slice object
stores a reference to the original PyStringObject * it is sliced from,
and the desired start and stop slice markers.  (It only supports step =
1.)  Its ob_sval is NULL until the string is rendered--but that rarely
happens!  Not only does this mean string slices are faster, but I bet
this generally reduces overall memory usage for slices too.

Now, one rule of the Python programming API is that "all strings are
zero-terminated".  That part of makes the life of a Python extension
author sane--they don't have to deal with some exotic Python string
class, they can just assume C-style strings everywhere.  Ordinarily,
this means a string slice couldn't simply point into the original
string; if it did, and you executed
  x = "abcde"
  y = x[1:4]
internally y->ob_sval[3] would not be 0, it would be 'e', breaking
the API's rule about strings.

However!  When a PyStringObject lives out its life purely within the
Python VM, the only code that strenuously examines its internals is
stringobject.c.  And that code almost never needs the trailing zero*. 
So I've added a new static method in stringobject.c:
    char * PyString_AsUnterminatedString(PyStringObject *)
If you call it on a lazy-evaluation slice object, it gives you back a
pointer into the original string's ob_sval.  The s->ob_size'th
element of this *might not* be zero, but if you call this function
you're saying that's a-okay, you promise not to look at it.  (If the
PyStringObject * is any other variety, it calls into PyString_AsString,
which renders string concatenation objects then returns ob_sval.)

Again: this behavior is *never* observed by anyone outside of
stringobject.c.  External users of PyStringObjects call
PyString_AS_STRING(), which renders all lazy concatenation and lazy
slices so they look just like normal zero-terminated PyStringObjects. 
With my patch applied, trunk still passes all expected tests.

Of course, lazy slice objects aren't just for literal slices created
with [x:y]. 
There are lots of string methods that return what are effectively
string slices, like lstrip() and split().

With this code in place, string slices that aren't examined by modules
are very rarely rendered.  I ran "pybench -n 2" (two rounds, warp 10
(whatever that means)) while collecting some statistics.  When it
finished, the interpreter had created a total of 640,041 lazy slices,
of which only *19* were ever rendered.


Apart from lazy slices, there's only one more enhancement when compared
with v1: string prepending now reuses lazy concatenation objects much
more often. There was an optimization in string_concatenate
(Python/ceval.c) that said: "if the left-side string has two
references, and we're about to overwrite the second reference by
storing this concatenation to an object, tell that object to drop its
reference".  That often meant the reference on the string dropped to 1,
which meant PyString_Resize could just resize the left-side string in
place and append the right-side.  I modified it so it drops the
reference to the right-hand operand too.  With this change, even with a
reduction in the allowable stack depth for right-hand recursion (so
it's less likely to blow the stack), I was able to prepend over 86k
strings before it forced a render.  (Oh, for the record: I ensure depth
limits are enforced when combining lazy slices and lazy concatenations,
so you still won't blow your stack when you mix them together.)


Here are the highlights of a single apples-to-apples pybench run, 2.6
trunk revision 52413 ("this") versus that same revision with my patch
applied ("other"):

Test minimum run-time    average 
run-time
 this    other   diff    this   
other   diff
---
 ConcatStrings:   204ms    76ms +168.4%   213ms    77ms
+177.7%
   CreateStringsWithConcat:   159ms   138ms  +15.7%   163ms  
142ms  +15.1%
 StringSlicing:   142ms    86ms  +65.5%   145ms   
88ms  +64.6%
---
Totals:  7976ms  7713ms   +3.4%  8257ms 
7975ms   +3.5%

I also ran this totally unfair benchmark:
    x = "abcde" * (2) # 100k characters
    for i in xrange(1000):
    y = x[1:-1]
and found my patched version to be 9759% faster.  (You heard that
right, 98x faster.)


I'm ready to post the patch.  However, as a result of this work, the
description on the original patch page is really no longer accurate:
   
http://sourceforge.net/tracker/index.php?func

Re: [Python-Dev] Massive test_sqlite failure on Mac OSX ... sometimes

2006-10-20 Thread skip
Following up on my earlier post...

I svn up'd both my g5 and my g4 powerbook (both running OSX 10.4.8, gcc
4.0.0 apple build 5026), built and tested both.  The test suite completed
fine on my powerbook, failed on the g5.  I tried running regrtest.py twice
more on the g5 with the -r flag.  It failed the first time, succeeded the
second.  I then made a series of run with the -f flag (thank you once again
for that Señor Peters).  I whittled it down to the following reliably
failing pair:

$ ./python.exe Lib/test/regrtest.py -l -f tests
test_ctypes
test_sqlite
test test_sqlite failed -- errors occurred; run in verbose mode for details
1 test OK.
1 test failed:
test_sqlite

For confirmation, this pair works fine on my g4 powerbook.  I've gone no
further so far.  It's bedtime.  Maybe someone else can at least try to
reproduce what I've come up with so far on other platforms or on another Mac
g5.

Skip
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] 2.4.4: backport classobject.c HAVE_WEAKREFS?

2006-10-20 Thread Fredrik Lundh
Larry Hastings wrote:

> I knocked out a prototype of this last week, emailed Mr. Lundh about it, 
> then forgot about it.

It's on my TODO list, so I haven't forgotten about it, but I've been (as 
usual) busy with other stuff.  I'll get there, sooner or later.

Posting this to the patch tracker and posting a note to the Py3K mailing 
list could be a good idea.



___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] The "lazy strings" patch

2006-10-20 Thread Talin
Interesting - is it possible that the same technique could be used to 
hide differences in character width? Specifically, if I concatenate an 
ascii string with a UTF-32 string, can the up-conversion to UTF-32 also 
be done lazily? If that could be done efficiently, it would resolve some 
outstanding issues that have come up on the Python-3000 list with 
regards to str/unicode convergence.

Larry Hastings wrote:
> 
> I've significantly enhanced my string-concatenation patch, to the point 
> where that name is no longer accurate.  So I've redubbed it the "lazy 
> strings" patch.
> 
> The major new feature is that string *slices* are also represented with 
> a lazy-evaluation placeholder for the actual string, just as 
> concatenated strings were in my original patch.  The lazy slice object 
> stores a reference to the original PyStringObject * it is sliced from, 
> and the desired start and stop slice markers.  (It only supports step = 
> 1.)  Its ob_sval is NULL until the string is rendered--but that rarely 
> happens!  Not only does this mean string slices are faster, but I bet 
> this generally reduces overall memory usage for slices too.
> 
> Now, one rule of the Python programming API is that "all strings are 
> zero-terminated".  That part of makes the life of a Python extension 
> author sane--they don't have to deal with some exotic Python string 
> class, they can just assume C-style strings everywhere.  Ordinarily, 
> this means a string slice couldn't simply point into the original 
> string; if it did, and you executed
>  x = "abcde"
>  y = x[1:4]
> internally y->ob_sval[3] would not be 0, it would be 'e', breaking the 
> API's rule about strings.
> 
> However!  When a PyStringObject lives out its life purely within the 
> Python VM, the only code that strenuously examines its internals is 
> stringobject.c.  And that code almost never needs the trailing zero*.  
> So I've added a new static method in stringobject.c:
>char * PyString_AsUnterminatedString(PyStringObject *)
> If you call it on a lazy-evaluation slice object, it gives you back a 
> pointer into the original string's ob_sval.  The s->ob_size'th element 
> of this *might not* be zero, but if you call this function you're saying 
> that's a-okay, you promise not to look at it.  (If the PyStringObject * 
> is any other variety, it calls into PyString_AsString, which renders 
> string concatenation objects then returns ob_sval.)
> 
> Again: this behavior is *never* observed by anyone outside of 
> stringobject.c.  External users of PyStringObjects call 
> PyString_AS_STRING(), which renders all lazy concatenation and lazy 
> slices so they look just like normal zero-terminated PyStringObjects.  
> With my patch applied, trunk still passes all expected tests.
> 
> Of course, lazy slice objects aren't just for literal slices created 
> with [x:y].  There are lots of string methods that return what are 
> effectively string slices, like lstrip() and split().
> 
> With this code in place, string slices that aren't examined by modules 
> are very rarely rendered.  I ran "pybench -n 2" (two rounds, warp 10 
> (whatever that means)) while collecting some statistics.  When it 
> finished, the interpreter had created a total of 640,041 lazy slices, of 
> which only *19* were ever rendered.
> 
> 
> Apart from lazy slices, there's only one more enhancement when compared 
> with v1: string prepending now reuses lazy concatenation objects much 
> more often. There was an optimization in string_concatenate 
> (Python/ceval.c) that said: "if the left-side string has two references, 
> and we're about to overwrite the second reference by storing this 
> concatenation to an object, tell that object to drop its reference".  
> That often meant the reference on the string dropped to 1, which meant 
> PyString_Resize could just resize the left-side string in place and 
> append the right-side.  I modified it so it drops the reference to the 
> right-hand operand too.  With this change, even with a reduction in the 
> allowable stack depth for right-hand recursion (so it's less likely to 
> blow the stack), I was able to prepend over 86k strings before it forced 
> a render.  (Oh, for the record: I ensure depth limits are enforced when 
> combining lazy slices and lazy concatenations, so you still won't blow 
> your stack when you mix them together.)
> 
> 
> Here are the highlights of a single apples-to-apples pybench run, 2.6 
> trunk revision 52413 ("this") versus that same revision with my patch 
> applied ("other"):
> 
> Test minimum run-timeaverage  run-time
> thisother   diffthisother   
> diff
> ---
>  
> 
> ConcatStrings:   204ms76ms +168.4%   213ms77ms 
> +177.7%
>   CreateStringsWithConcat:   159ms   138ms  +15.7%   163ms   142ms  
> +15.1%
> Stri