There's not enough information to guess the "real problem", but it could be this:
"variable size" objects (declared with PyObject_VAR_HEAD) are limited to
INT_MAX items since the ob_size field is declared as 'int'.
This means that a Python string, tuple, or list (among other types) may
be limited to about 2 billion items on ILP32 and LP64 architectures.
Dicts and probably sets are also limited to INT_MAX elements, because
the "ma_fill" and "ma_used" fields are ints.
If you don't mind recompiling all your extensions, you could change the
type of ob_size to long in the "#define PyObject_VAR_HEAD". I don't
know what breaks when you do this, but maybe a few google or google
groups searches could help you find others who have tried this.
Jeff
PS the limit of 500MB of "real data" in the 32-bit system may be because
a realloc may temporarily require (old size + new size) storage when it
does the equivalent of
new_ptr = malloc(new_size)
memcpy(new_ptr, old_ptr, old_size)
free(old_size)
which will temporarily use >900MB of data when resizing a ~450MB object.
Python frequently uses realloc() to grow structures like lists. If you
are working with strings, then
s = s + t
doesn't ever use realloc (in 2.3.x anyway) but always allocates and
fills the result 's+t', only freeing the old value of s later when it is
no longer reacahable (which could be as soon as the assignment statement
completes)
pgp7QRCaTXJkk.pgp
Description: PGP signature
-- http://mail.python.org/mailman/listinfo/python-list
