Ok thanks for the advice everyone. Cheers Pete
On Thu, Aug 23, 2012 at 10:58 AM, Jerry Hill <malaclyp...@gmail.com> wrote: > On Wed, Aug 22, 2012 at 5:23 PM, Pete O'Connell <pedrooconn...@gmail.com> > wrote: >> OK maybe I am wrong about it being slow (I thought for loops were >> slower than lis comprehensions). But I do know I need it to be as fast >> as possible if I need to run it on a thousand files each with hundreds >> of thousands of lines > > You're quite likely wrong about that. The overall runtime of your > code is very likely to end up dominated by I/O with the hard drive, > not the microsecond differences in how you process each line. The way > to know for sure is to write code that is easy to read and does the > correct thing. Then run the code an see if it's too slow for your > purposes. If it is, profile the code and see where it's spending all > of its time. Only then can you actually optimize the right places. > > Once you've identified where your code needs to be faster, you're > likely to get more mileage out of a clever algorithm change than a > micro-optimization to your filtering. Or maybe you'll discover that it > really is worthwhile to optimize your filtering, because it's inside a > tight loop, and that's where all of your time is spent. There are a > lot of tricks you can use to slightly speed up your python code, but > none of them are worth doing until you know for sure that you're > optimizing in the right place. And if it turns out that the code's > runtime is fine as first written, you don't have to waste your time > doing lots of work for little gain. > > -- > Jerry > _______________________________________________ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > http://mail.python.org/mailman/listinfo/tutor -- - _______________________________________________ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
