Re: should writing Unicode files be so slow

2010-03-21 Thread Ben Finney
djc writes: > I shall concentrate on finding an optimal time to upgrade from Python > 2.6. Note that Python 2.7, though nearly ready, is not yet released http://www.python.org/download/releases/>. -- \“… Nature … is seen to do all things Herself and through | `\ herself

Re: should writing Unicode files be so slow

2010-03-21 Thread djc
Antoine Pitrou wrote: > Le Fri, 19 Mar 2010 17:18:17 +, djc a écrit : >> changing >> with open(filename, 'rU') as tabfile: to >> with codecs.open(filename, 'rU', 'utf-8', 'backslashreplace') as >> tabfile: >> >> and >> with open(outfile, 'wt') as out_part: to >> with codecs.open(outfile, 'w', '

Re: should writing Unicode files be so slow

2010-03-19 Thread Antoine Pitrou
Le Fri, 19 Mar 2010 17:18:17 +, djc a écrit : > > changing > with open(filename, 'rU') as tabfile: to > with codecs.open(filename, 'rU', 'utf-8', 'backslashreplace') as > tabfile: > > and > with open(outfile, 'wt') as out_part: to > with codecs.open(outfile, 'w', 'utf-8') as out_part: > > ca

Re: should writing Unicode files be so slow

2010-03-19 Thread Ben Finney
"Gabriel Genellina" writes: > Ok. Your test program is too large to determine what's going on. Try > to determine first *which* part is slow: Right. This is done by the diagnostic technique of writing *new*, minimal, complete programs that exercise each piece of the functionality separately. Yo

Re: should writing Unicode files be so slow

2010-03-19 Thread Gabriel Genellina
En Fri, 19 Mar 2010 14:18:17 -0300, djc escribió: Ben Finney wrote: What happens, then, when you make a smaller program that deals with only one file? What happens when you make a smaller program that only reads the file, and doesn't write any? Or a different program that only writes a file,

Re: should writing Unicode files be so slow

2010-03-19 Thread djc
Ben Finney wrote: > What happens, then, when you make a smaller program that deals with only > one file? > > What happens when you make a smaller program that only reads the file, > and doesn't write any? Or a different program that only writes a file, > and doesn't read any? > > It's these sort

Re: should writing Unicode files be so slow

2010-03-19 Thread Ben Finney
djc writes: > Ben Finney wrote: > > Could you please: > > > > * simplify it further: make a minimal version that demonstrates the > > difference you're seeing, without any extraneous stuff that doesn't > > appear to affect the result. > > > > * make it complete: the code you've shown doesn'

Re: should writing Unicode files be so slow

2010-03-19 Thread djc
Ben Finney wrote: > djc writes: > >> I have a simple program to read a text (.csv) file > > Could you please: > > * simplify it further: make a minimal version that demonstrates the > difference you're seeing, without any extraneous stuff that doesn't > appear to affect the result. > > * m

Re: should writing Unicode files be so slow

2010-03-18 Thread Ben Finney
djc writes: > I have a simple program to read a text (.csv) file Could you please: * simplify it further: make a minimal version that demonstrates the difference you're seeing, without any extraneous stuff that doesn't appear to affect the result. * make it complete: the code you've shown

should writing Unicode files be so slow

2010-03-18 Thread djc
I have a simple program to read a text (.csv) file and split it into several smaller files. Tonight I decided to write a unicode variant and was surprised at the difference in performance. Is there a better way? > from __future__ import with_statement > import codecs > > def _rowreader(filename,