Ravi Kondamuru wrote:
Denis, These are 32bit, 64bit counters (essentially numbers).
Bob, There are well over 10K counters in the log file that are updated
every 5 secs. If a counter1's graph was requested, log will have to be
parsed once to get the data points. If a user asked for a counter2,
now it needs to be retrieved from the log file. Which essentially
means having to go through the logfile again. This is because i want
to try to avoid using a database to store values after parsing the file.
Here is a little test program I wrote to check timing on a file of
*100,000* 64 bit counters.
import time, struct
ctrs = 100000
s = time.time()
# create a file of 800,000 characters (100,000 64 bit numbers)
f = open('ravi.dat', 'wb')
for i in range(ctrs):
f.write('abcdefgh') # it does not matter what we write as long as it
is 8 bytes.
print time.time() - s
s = time.time()
l = []
# read the file
f = open('ravi.dat', 'rb').read()
# unpack each 8 characters into an integer and collect in a list
for b in range(0, ctrs, 8):
n = struct.unpack('q', f[b:b+8])
l.append(n)
print time.time() - s
Writing the file took 0.14 seconds on my computer
Reading and unpacking 0.04 seconds.
I think you can set performance issues aside.
There is a principle in programming that proposes:
1 - get a program running. Preferably in Python as development time is
much faster.
2 - check its performance.
3 - refine as needed.
--
Bob Gailer
Chapel Hill NC
919-636-4239
_______________________________________________
Tutor maillist - Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor
