Is there some methods like 'datareader' in MySQLdb for handling mass data
hi all: when I handled mass data table in mysql with python's MySQLdb lib, something frustrated me. I could't find any function like datareader, which yield one row by recording rows' anchor ,after each time I invoked 'read()' in a loop.fetchall or fetchmany just fetched all data in once, which take so much memory when meeting one mass data table. Is anybody know that function or other lib similar to MySQLdb ? thanks! -- http://mail.python.org/mailman/listinfo/python-list
Re: Is there some methods like 'datareader' in MySQLdb for handling mass data
On Jan 6, 7:29 pm, Bruno Desthuilliers wrote: > Jeremy.Chen a écrit : > > > hi all: > > when I handled mass data table in mysql with python's MySQLdb lib, > > something frustrated me. I could't find any function like datareader, > > which yield one row by recording rows' anchor ,after each time I > > invoked 'read()' in a loop. > > I think you want cursor.fetchone(). But read below... > > > fetchall or fetchmany just fetched all data > > in once, which take so much memory when meeting one mass data > > table. > > The cursor object is it's own iterator. So you just have to do: > > cursor.execute(your_query) > for row in cursor: > process(row) > > HTH That's what I want. I see,cursor also can be a iterator. -- http://mail.python.org/mailman/listinfo/python-list
Re: ftp seems to get a delayed reaction.
On Jan 6, 7:56 pm, Antoon Pardon wrote:
> I have managed to prune the script, so I can post it here:
>
> ==
>
> from ftplib import FTP
>
> bckrt = "/tmpstor/rcpc42"
>
> def process():
> print "making directory"
> try:
> ftp.mkd('ftp-tst')
> print "mkdir succeeded"
> except Exception, ErrMsg:
> print "mkdir failed: %s" % ErrMsg
> fl = open("tstfile")
> print "storing file"
> ftp.storbinary("STOR ftp-tst/ftp-file\n", fl)
> fl.close()
>
> ftp = FTP('ftphost', 'user', 'passwd')
> ftp.set_pasv(False)
> ftp.cwd(bckrt)
>
> print "Doing once"
> process()
> print "Doing twice"
> process()
>
> --
> And this is the output:
> ==
> Doing once
> making directory
> mkdir succeeded
> storing file
> Doing twice
> making directory
> mkdir failed: 500 '': command not understood.
> storing file
> Traceback (most recent call last):
> File "ftptst", line 24, in
> process()
> File "ftptst", line 14, in process
> ftp.storbinary("STOR ftp-tst/ftp-file\n", fl)
> File "/usr/lib/python2.5/ftplib.py", line 425, in storbinary
> self.voidcmd('TYPE I')
> File "/usr/lib/python2.5/ftplib.py", line 246, in voidcmd
> return self.voidresp()
> File "/usr/lib/python2.5/ftplib.py", line 221, in voidresp
> resp = self.getresp()
> File "/usr/lib/python2.5/ftplib.py", line 216, in getresp
> raise error_perm, resp
> ftplib.error_perm: 521 "/tmpstor/rcpc42/ftp-tst" directory exists
ftp.storbinary("STOR ftp-tst/ftp-file\n", fl)
--
I think the params after STOR should't be a path,should be splited.
ftp.cwd("ftp-tst")
ftp.storbinary("STOR ftp-file\n", fl)
--
http://mail.python.org/mailman/listinfo/python-list
Re: Regex Generator From Multiple Files
On Jan 6, 8:48 am, MRAB wrote:
> James Pruitt wrote:
> > I am looking for a way given a number of files, say 3, that represent
> > technical support tickets in the same format to generate regular
> > expressions for the different fields automatically.
>
> > An example from of one line from each file:
> > Date: 12/30/2008 Room: 457 Building: Main
> > Date: 12/31/2008 Room: A21 Building: Annex
> > Date: 1/4/2009 Room: L69 Building: Library
>
> > The program would then, possibly using the python diff library, generate
> > the regular expression needed to parse out different fields. In this
> > case it might return a tuple like
> > ("^Date:[\w]+(.*)[\w]+Room","Room:[\w]+(.*)[\w]+Building","Building:[\w]+(.*)[\w]+$")
> > that would match each of the fields based on the common data and sort of
> > assume that what doesn't change between them is data we are looking for.
>
> Why not just assume that each field consists of a word terminated by a
> colon, then some text, then the next field or the end of the line?- Hide
> quoted text -
>
> - Show quoted text -
do you mean the sub method?
-
re.sub( r'(?i)(example)',self.captureRegxp,content )
--
http://mail.python.org/mailman/listinfo/python-list
