On Mon, 01 Feb 2010 14:04:01 -0800, Jonathan Gardner wrote: > Having worked with datasets that are truly enormous, whenever I see a > list that is unbounded (as this appears to be), I freak out because > unbounded can be a really, really big number. Might as well write > software that works for the truly enormous case and the really large > case and be done with it once and for all.
Then write it to work on iterators, if at all possible, and you won't care how large the data stream is. Even databases have limits! Of course you should write your code to deal with your expected data, but it is possible to commit overkill as well as underkill. There are trade- offs between development effort, runtime speed, memory usage, and size of data you can deal with, and maximizing the final one is not always the best strategy. Look at it this way... for most applications, you probably don't use arbitrary precision floats even though floating point numbers can have an unbounded number of decimal places. But in practice, you almost never need ten decimal places to pi, let alone an infinite number. If the creators of floating point libraries wrote software "that works for the truly enormous case", the overhead would be far too high for nearly all applications. Choose your algorithm according to your expected need. If you under- estimate your need, you will write slow code, and if you over-estimate it, you'll write slow code too. Getting that balance right is why they pay software architects the big bucks *wink* -- Steven -- http://mail.python.org/mailman/listinfo/python-list
