[issue3063] memory leak in random number generation

Amaury Forgeot d'Arc added the comment: For the record, this was finally fixed with issue2862: gc.collect() now clears the free-lists during the collection of the highest generation. -- nosy: +amaury.forgeotdarc ___ Python tracker

[issue3063] memory leak in random number generation

Tim Peters <[EMAIL PROTECTED]> added the comment: Float objects also require, as do all Python objects, space to hold a type pointer and a reference count. So each float object requires at least 16 bytes (on most 32-bit boxes, 4 bytes for the type pointer, 4 bytes for the refcount, + 8 bytes for

[issue3063] memory leak in random number generation

Facundo Batista <[EMAIL PROTECTED]> added the comment: Grant, A float takes 64 bits. 100 million floats take 800 MB, *just* the floats. You're also building a list of 100 million places. Maybe you shouldn't be building this structure in memory? In any case, you should raise this issue in comp.

[issue3063] memory leak in random number generation

Grant Tang <[EMAIL PROTECTED]> added the comment: Facundo: I understand now. You mean every unique float number used will be an object in memory. And never been released until Python quit. Is there any way to reclaim these memory? We need 3G memory to create a list of 100million randum numbers.

[issue3063] memory leak in random number generation

Facundo Batista <[EMAIL PROTECTED]> added the comment: So, 0.0 would be cached, and the 414m+384m would be from the list itself, right? I tried, >>> data = [(1.0/i) for i in xrange(1,1)] And the memory consumption was the big one. Grant, the 800 MB is taken by ONE 0.0, and a list of zi

[issue3063] memory leak in random number generation

Grant Tang <[EMAIL PROTECTED]> added the comment: Here I am confused. 100million floats in a list takes about 800M byte memory. This is acceptable. for i in xrange(1): data[i] = random() so it should be 800M plus a float returned by random(). But the problem is after this loop, e

[issue3063] memory leak in random number generation

Tim Peters <[EMAIL PROTECTED]> added the comment: They stayed alive simultaneously because you stored 100 million of them simultaneously in a list (data[]). If instead you did, e.g., for i in xrange(1): x = random() the problem would go away -- then only two float objects are simul

[issue3063] memory leak in random number generation

Grant Tang <[EMAIL PROTECTED]> added the comment: I agree with Tim's comment. The problem's why these floats keep alive even after random() call returns. Then this becomes a garbage collection issue? ___ Python tracker <[EMAIL PROTECTED]>

[issue3063] memory leak in random number generation

Tim Peters <[EMAIL PROTECTED]> added the comment: Strongly doubt this has anything to do with random number generation. Python maintains a freelist for float objects, which is both unbounded and immortal. Instead of doing "data[i] = random()", do, e.g., "data[i] = float(s)", and I bet you'll se

[issue3063] memory leak in random number generation

Facundo Batista <[EMAIL PROTECTED]> added the comment: Confirmed the issue in the trunk right now: (the number between square brackets point to the 'top' information below) [EMAIL PROTECTED]:~/devel/reps/python/trunk$ ./python Python 2.6a3+ (trunk:64009, Jun 7 2008, 09:51:56) [GCC 4.2.3 (Ubu

[issue3063] memory leak in random number generation

New submission from Grant Tang <[EMAIL PROTECTED]>: #the following code consume about 800M memory, which is normal n = 1 data = [0.0 for i in xrange(n)] #however, if I assign random number to data list, it will consume extra 2.5G memory. from random import random for s in xrange(n):