Terry Reedy <tjre...@udel.edu> writes:
> > Right. And if the file is large enough that even parsing all the
> > records to check the fields takes hours, well, that's where I'm at.
>
> So what is the problem? Let it run overnight.
The idea of buying faster and faster computers is to not have to wait
overnight. The sooner you can get the results (preferably within
minutes not hours), the sooner they can be used in the next task,
which depends on them. Language implementation affects program speed
just as hardware does, thus the grumbling about Python being slow.
> If you want more
> speed, try numpy or Cython or C functions (swigged or ctyped) or...
I should try out Cython. Using plain C would slow down development an
awful lot. Either way would require putting most of the logic in C to
get the best speedup, since it would require building
Python-unfriendly data structures (the natural Pythonic structures
use a huge amount of dict lookups and storage allocation/release).
> for line in open('humongous.dat', 'r'):
> try:
> <parse line>
> <check fields and update summary>
> except Exception as e:
> print(line, e)
>
> (How trivial in Python!)
It's a little more complicated than that since multiple threads are
involved, and "except Exception" is considered an antipattern, but
yeah, I'm already doing stuff like this.
--
http://mail.python.org/mailman/listinfo/python-list
