urllib2 script slowing and stopping
I have a small script that reads several CSV files in a directory and
puts the data in a DB using Django.
There are about 1.7 million records in 120 CSV files, I am running the
script on a VPS with about 512mb of memory python 2.6.5 on ubuntu
10.04.
The script gets slow and seems to lock after about 87 records.
running top show that the memory is all being used up y the python
process, is there someway I can improve on this script?
class Command(BaseCommand):
def handle(self, *args, **options):
count = 0
d = urllib2.urlopen(postcode_dir).read()
postcodefiles = re.findall('', d)
nprog = 0
for n in range(nprog, len(postcodefiles)):
fl = postcodefiles[n]
print 'Processing %d %s ...' % (n, fl)
s = urllib2.urlopen(postcode_dir + fl)
c = csv.reader(s.readlines())
for row in c:
postcode = row[0]
location = Point(map(float, row[10:12]))
Postcode.objects.create(code=postcode,
location=location)
count += 1
if count % 1 == 0:
print "Imported %d" % count
s.close()
nprog = n+1
Thanks
-Dan
--
http://mail.python.org/mailman/listinfo/python-list
Re: urllib2 script slowing and stopping
On Oct 11, 10:07 pm, Ian wrote: > On Oct 11, 2:48 pm, Dantium wrote: > > > I have a small script that reads several CSV files in a directory and > > puts the data in a DB using Django. > > > There are about 1.7 million records in 120 CSV files, I am running the > > script on a VPS with about 512mb of memory python 2.6.5 on ubuntu > > 10.04. > > > The script gets slow and seems to lock after about 87 records. > > running top show that the memory is all being used up y the python > > process, is there someway I can improve on this script? > > Probably you have "DEBUG = True" in your Django settings.py file. In > debug mode, Django records every query that is executed in > django.db.connection.queries. To fix it, either disable debug mode or > just have your script go in and clear out that list from time to time. > > HTH, > Ian Yeah thanks that helped! It was still running really low on memory by the end though but they all got added. -- http://mail.python.org/mailman/listinfo/python-list
