On 11/05/14 08:56, Ross Gayler wrote:
It looks to me as though 32 and 64 bit versions of Python on 64 bit
Windows are both really 32 bit Python, differing only in how they
interact with Windows.
No! Pointers are 64 bit, Python integers (on Python 2.x) are 32 bit.
Microsoft decided to use a 32 bit long in MSVC for backwards
compatiblity, but also because the AMD64 (x86-64) architecture was
designed to use a 64 address with a 32 bit offset. (A 64 bit long was
originally slightly less efficient.) You can see the value of 64 bit
Python e.g. if you allocate a lot of objects or if you try to mmap a
huge file. With 32 bit Python you are limited to only 2 GB of virtual
memory. In 64 bit Python you can in practice mmap as much as you want.
The element size of what you try to index also matters. While a C long
and a Python int is 32 bit on Windows, 64-bit Python will use a 64-bit
offset internally (Py_ssize_t and Py_intptr_t) even on Windows. The 32
bit Python int just limits how many objects you can index from Python
space before Python roll over to using long instead of int. It does not
limit the amount of memory a Python int can index. In is only when you
index an array of bytes you will see the roll-over from Python int to
Python long at the 2 GB limit. Typically, object will be much larger
than one byte.
Here are two examples:
- A one-dimensional NumPy array with dtype np.float64 can keep 16 GB of
data before a 32 bit index is too small and Python starts to use long.
A two-dimensional NumPy array with dtype np.float64 can keep 256 GB of
data before a 32 bit index is too small.
- A Python list stores internally an array of pointers, each of which is
64 bit. So just indexing those goes up to 16 GB of pointer data before
the int rolls over. Then each of these pointers point to a Python
object. A Python float on my computer (not Windows) is 24 bytes, which I
got from sys.getsizeof(1.) So 2**32 of those are another 383 GB. So if I
indexed a list of Python floats on this computer, Python could handle an
almost 400 GB data structure with a 32 bit int as indexer without
rolling over to long.
This is obviously way beyond anything the 2 GB limit on 32 bit Python
allows.
Sturla
--
https://mail.python.org/mailman/listinfo/python-list
