poostenr added the comment:
Eric, Steven,
During further testing I was not able to find any real evidence that the
statement I was focused on had a real performance issue.
As I did more testing I noticed that appending data to the file slowed down.
The file grew initially with ~30-50KB increments and around 500KB it had slowed
down to ~3-5KB/s, until around 1MB the file grew at ~1KB/s. I found this to be
odd and because Steven had mentioned other processes, I started looking at some
other statements.
After quite a lot of trial and error, I was able to use single quotes and
increase my performance to acceptable levels.
Example 3 below is how I resolved it.
Can you explain to me why there was a performance penalty in example 2 ?
Python did something under the hood that I am overlooking.
Did conv.escape_string() change something about columnvalue, so that adding a
single quote before and after it introduced some add behavior with writing to
file ? I am not an expert on Python and remember reading something about
Dynamic typing.
Example 1: Fast performance, variable s is not encapsulated with single quotes
6.5MB parsed in ~1 minute.
for key in listkeys:
keyvalue = self.recordstats[key]
fieldtype = keyvalue[0]
columnvalue = record[key]
columnvalue = conv.escape_string(columnvalue)
if (count > 1):
s = "{0},".format(columnvalue) # No single quotes
else
s = "{0},".format(columnvalue) # No single quotes
count -= 1
Append s to file.
Example 2: Slow performance, pre- and post-fixed variable s with single quotes
6.5MB parsed in 35 minutes.
for key in listkeys:
keyvalue = self.recordstats[key]
fieldtype = keyvalue[0]
columnvalue = record[key]
columnvalue = conv.escape_string(columnvalue)
if (count > 1):
s = "'{0}',".format(columnvalue) # Added single quotes
else
s = "'{0}',".format(columnvalue) # Added single quotes
count -= 1
Append s to file.
Example 3: Fast performance, variable columnvalue is pre- and post-fixed with
single quotes
6.5MB parsed in !45 seconds.
for key in listkeys:
keyvalue = self.recordstats[key]
fieldtype = keyvalue[0]
columnvalue = record[key]
columnvalue = conv.escape_string("'" + columnvalue + "'") # Moved single
quotes to this statement.
if (count > 1):
s = "{0},".format(columnvalue)
else
s = "{0},".format(columnvalue)
count -= 1
Append s to file.
----------
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue26118>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
