[runes] > You have sets in Python 2.3x and 2.4x. I don't know if they can handle > your amounts of data, but i guess so.
The original poster (OP) spoke about 10**13 numbers, if I remember correctly (I did not keep the original message). It all depends on the real sparsity of the spare sets! :-) If not sparse enough, this might be a bit inordinate for standard Python sets on most machines, also given that each Python integer uses a lot more than 4 bytes... On the other hand, the OP also wrote that the numbers he handles are grouped in big and compact intervals, separated by big unpopulated holes. So maybe his data could be represented in Python as a user-written type hiding a sorted list of intervals, for which it should be relatively easy to write set functions. That might make the problem tractable enough, with reasonably speedy processing. As last resort, the wanted sets could be represented as sorted files of numbers. Slower, but set operations would also be easy to write. -- François Pinard http://pinard.progiciels-bpi.ca -- http://mail.python.org/mailman/listinfo/python-list
